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ABSTRACT  (U).  This  paper  describes  the  methodologies  and  implementations  used  by 
the  Joint  Advanced  Distributed  Simulation  (JADS)  Electronic  Warfare  (EW)  Test  team  to 
establish  baseline  performance  data  from  testing  at  the  open  air  range  (OAR)  and  a 
hardware-in-the-loop  (HITL)  facility,  the  U.S.  Air  Force  Electronic  Warfare  Evaluation 
Simulator  (AFEWES).  The  AFEWES  laboratory’s  real-time,  actual-frequency  simulations 
of  radio  frequency  (RF)  guided  surface-to-air-missiles  (SAMs)  and  anti-aircraft  artillery 
(AAA)  projectiles  were  used  to  supplement  data  shortfalls  from  OAR  threat  system  testing 
of  an  aircraft  self-protection  EW  suite.  Key  points  of  discussion  include  1) 
gathering/evaluating  OAR  repeatable  baseline  data;  2)  gathering/evaluating  HITL 
repeatable  baseline  data;  and  3)  building  JADS  EW  Test  end-to-end  scenarios.  The  Air 
Force  Flight  Test  Center,  Edwards  Air  Force  Base,  California,  manages  both  the  OAR  and 
AFEWES;  the  later  is  located  at  Air  Force  Plant  Number  4  in  Fort  Worth,  Texas. 

1.0  INTRODUCTION  (U).  The  JADS  EW  Test  program  is  an  Office  of  the  Secretary 
of  Defense-sponsored  triservice  effort  designed  to  determine  how  well  an  emerging 
technology,  advanced  distributed  simulation  (ADS),  can  be  used  to  support  EW  test  and 
evaluation  (T&E).  There  are  three  phases  for  this  effort  and  all  phases  are  complete. 
However,  the  focus  of  this  paper  is  on  Phase  1  and  how  those  results  impacted  our  overall 
test  design.  The  Phase  1  original  purpose  was  to  use  an  OAR  flight  test  to  establish  the 
baseline  performance,  to  provide  sufficient  information  to  recreate  the  OAR  environment 
using  ADS,  and  to  validate  the  data  from  the  ADS  environment  using  the  OAR  data.  The 
baseline  performance  was  characterized  using  standard  measures  of  performance  and 
measures  of  effectiveness.  These  measures  will  be  correlated  to  measure  the  enhancement 
or  degradation  in  performance  when  using  multiple  T&E  facilities  linked  together  via 
long-haul  telecommunications  during  the  ADS-based  test  phases.  Integral  to  this 
approach  was  understanding  the  differences  between  the  OAR  and  at  AFEWES  facility 
differences.  This  paper  discusses  what  the  JADS  team  had  to  do  to  isolate  ADS 
differences  from  facility  differences. 

2.0  BASELINE  COLLECTION  (U).  The  OAR  flight  test  was  accomplished  in  two 
parts.  Both  Part  1  (risk  reduction)  and  Part  2  (baseline  collection)  OAR  flights  used  an  F- 
16  equipped  with  a  self-protection  jamming  pod  flying  against  ground-based  SAM  and 
AAA  sites.  Risk  reduction  was  a  series  of  six  flights  designed  to  ensure  success  in  the 
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baseline  collection.  Our  goal  in  OAR  baseline  collection  was  to  collect  sufficient  data 
from  a  single  reference  test  condition  to  characterize  ten  different  measures  of 
performance  (MOP)  populations.  A  statistical  comparison  of  the  baseline  populations  to 
those  produced  in  the  ADS  environment  was  expected  to  demonstrate  the  extent  that  the 
ADS  environment  changed  the  OAR  results.  A  number  of  approaches  were  tested  and 
refined  in  the  risk  reduction  flights  allowing  JADS  to  minimize  trial-to-trial  variance  and  to 
characterize  the  instrumentation  available  on  the  range.  Ultimately,  a  wing-level,  fixed- 
altitude  flight  profile  with  fixed  end  points  was  selected  to  reduce  pilot  induced  variation. 
The  jammer  pod  software  load  was  created  to  minimize  false  alarms  and  to  provide  a 
single  response  technique  per  threat.  The  threat  operator  rules  of  engagement  were 
established  to  reduce  operator  induced  variance  and  to  ensure  the  engagements  could  be 
recreated  in  the  distributed  environment.  A  single  pod,  a  single  aircraft  and  a  standard 
aircraft  load  were  selected.  Instrumentation  was  characterized  and  JADS  elected  to  add 
an  additional  HITL  test  to  the  baseline  data  collection. 

(U)  The  OAR  risk  reduction  flights  proved  the  available  instrumentation  was  not 
sufficient  to  collect  all  the  measures,  and  it  was  insufficient  to  allow  several  known 
variance  sources  to  be  separated  from  ADS  errors  in  the  later  ADS-based  phases  of  the 
test.  The  instrumentation  available  for  measuring  the  time  the  pod  needed  to  correctly 
identify  the  threat  and  to  respond  could  not  be  synchronized  to  any  clock  source  making  it 
impossible  to  measure  reliably.  Ground  instrumentation  could  not  be  used  to  overcome 
this  problem.  In  addition,  the  range  instrumentation  was  not  sufficient  to  completely 
monitor  and  characterize  unintended  RF  emissions.  The  range  receivers  were  manually 
tuned  making  it  difficult  to  observe  and  quickly  capture  transient  signals  in  the  range 
environment.  The  pod  was  programmed  to  ignore  most  of  these  signals  to  help 
compensate  for  this  shortfall.  While  this  helped  overcome  some  variance  sources  other 
variance  sources  remained.  These  are  listed  in  the  figure  below. 


Test  Component 


_ F-16  radar  cross  section  (RCS) _ 

_ ALQ-131  antenna  patterns _ _ 

_ Threat  representation  _ 

Electronic  countermeasures  (ECM)  technique 

_ Missile  flyout _ 

Time-space-position  information  (TSPI)/attitude 
RF  environment  _ 


3.0  HITL  TEST  APPROACH  (U).  The  JADS  EW  Test  integrated  product  team  (IPT) 
met  in  early  March  1998  and  determined  the  best  approach  to  meeting  the  data  shortfall. 
The  IPT  considered  several  alternatives  to  collect  the  missing  data;  however,  the  winning 
approach  was  to  conduct  a  hardware-in-the-loop  test  at  AFEWES.  This  alternative 
provided  a  natural  bridge  for  aggregating  the  known  non- ADS  variance  sources  and 
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looking  at  their  impact  in  a  single  test  event.  The  approach  was  to  execute  the  test  and 
use  the  same  correlation  process  to  determine  the  impact  of  the  variances  on  the  ten 
MOPs.  In  order  for  this  “unscheduled”  event  to  occur,  multiple  challenges  were 
immediate:  1)  EW  hardware  and  maintenance  personnel  from  the  Air  National  Guard  Air 
Force  Reserve  Test  Center  (AATC)  in  Tucson,  Arizona,  would  need  to  be  moved  from 
AATC  to  AFEWES  in  Fort  Worth,  Texas;  and  2)  Funds  would  need  to  be  identified  and 
put  on  contract  in  time  for  a  July  HITL  test.  These  challenges  were  met  and  the  test  was 
executed  on  schedule.  The  following  paragraphs  discuss  the  test  assets  and  execution  of 
the  HITL  test. 

3.1  General  AFEWES  Overview  (U).  The  AFEWES  laboratory  contains  simulations 
of  a  number  of  threat  weapon  systems  along  with  the  instrumentation  and  monitoring 
resources  necessary  to  allow  realistic,  relative  effectiveness  evaluation  of  electronic 
combat  equipment.  The  weapon  system  simulations  employ  electronic  circuitry  and 
components  modeled  after  those  employed  in  actual  systems  to  provide  a  real-time  RF 
interface  with  the  electronic  combat  equipment.  Display  interfaces  are  simulated  to 
provide  authentic  man-in-the-loop  responses.  Free-space  signal  propagation  effects, 
antenna  systems,  aircraft  characteristics,  and  weapons  parameters  are  modeled  within 
digital  computer  programs.  These  computers  and  models  are  interfaced  with  the  RF 
hardware  to  produce  the  complete  simulation  environment. 

3.2  Threats  (U).  The  closed-loop  threats  for  this  effort  included  the  Simulated  Air 
Defense  System  (SADS)  III,  SADS  VIM,  AAA  and  SADS  VIII.  Antenna  patterns  for 
threat  radars,  as  well  as  for  the  system  under  test  (SUT)  were  loaded  into  computer  look¬ 
up  tables.  A  brief  overview  of  the  four  AFEWES  threat  simulators  is  provided  in  the 
following  paragraphs. 

3.2.1  AFEWES  SADS  HI  (U).  The  SADS  III  simulation  was  three  different  simulations 
running  in  real  time  during  dynamic  testing.  The  first  simulation  was  the  threat  target¬ 
tracking  radar  with  scope  displays  and  man-in-the-loop  interfaces.  The  second  simulation 
was  that  of  the  missile  associated  with  the  SADS  III.  The  third  simulation  component  was 
the  penetrating  aircraft.  The  threat  target-tracking  radar  simulation  was  interfaced  with 
the  SUT  at  RF.  The  missile  simulation,  which  is  primarily  a  computer  simulation,  was 
interfaced  with  the  SUT  by  sending  a  RF  up-link  signal. 

3.2.2  AFEWES  SADS  VIII  (U).  The  SADS  VIII  simulation  was  four  different 
simulations  running  in  real  time  during  dynamic  testing.  The  first  simulation  was  the 
threat  target-acquisition  radar  with  displays  and  man-in-the-loop  interfaces.  The  second 
simulation  was  the  threat  target-tracking  radar;  it  too  had  a  radar  scope  display  and  man- 
in-the-loop  interfaces.  The  third  simulation  was  the  missile  associated  with  the  SADS 
VIII.  The  fourth  simulation  component  was  the  penetrating  aircraft.  The  threat  target- 
acquisition  radar  and  target-tracking  radar  simulations  were  interfaced  with  the  SUT  at 
RF.  The  missile  simulation,  which  is  primarily  a  computer  simulation,  was  interfaced  with 
the  SUT  by  sending  a  RF  up-link  signal. 
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3.2.3  AFEWES  SADS  VIM  (U).  The  SADS  VI  RF  missile  simulator  provided  a  high 
fidelity  target  generation  through  signal  injection  immediately  behind  the  antenna  feed 
assembly  at  actual  RF  wavelength  and  at  the  correct  real-time  amplitude  and  phase.  The 
target  generation  system  provided  a  highly  accurate  signal  fidelity  including  the  effects  of 

Target  (victim  aircraft)  cross  section 

Range  attenuation 

Antenna  patterns 

Illuminator  transmitter  antenna 
ECM  receiver  antexma(s) 

ECM  transmitter  antenna(s) 

-  Missile  receiver  antenna  patterns  (including  sum,  pitch,  yaw  and  rear  reference) 

Transmitter  power  levels 

Insertion  losses 

Doppler  frequencies 

-  Target-tracking  radar  (TTR)-to-target 

-  Target-to-missile 

-  TTR-to-missile 

Time  of  arrival 

(U)  In  multiple  target  test  environments  each  target  is  generated  independently  for  each 
of  the  RF  paths  (i.e.,  TTR-to-target,  target-to-missile,  and  TTR-to-missile)  and  the  signals 
are  combined  in  passive  RF  combiners  prior  to  injection  into  the  appropriate  antenna  port. 
This  is  the  same  method  of  signal  combination  that  occurs  at  the  antenna  in  a  free-space 
environment.  All  RF  modulation  is  calculated  in  real  time  at  the  simulation  interval,  and 
the  levels  and  frequencies  are  derived  from  the  instantaneous  solution  of  the  appropriate 
radar  range  equation. 

(U)  The  position  and  velocities  of  the  target  and  the  missile  are  monitored  and  recorded 
in  real  time  and  vector  miss  distance  is  determined  for  each  target  when  the  point  of 
closest  approach  is  reached  for  that  target. 

3.2.4  AFEWES  Anti-Aircraft  Artillery  (AAA)  (U).  The  AAA  simulation  was  four 
different  simulations  running  in  real  time  during  dynamic  testing.  The  first  simulation  was 
the  threat  target-acquisition  radar  with  displays  and  man-in-the-loop  interfaces.  The 
second  simulation  was  the  threat  target-tracking  radar;  it  too  had  a  radar  scope  display 
and  man-in-the-loop  interfaces.  The  third  simulation  was  the  AAA  projectiles.  The  fourth 
simulation  component  was  the  penetrating  aircraft.  The  threat  target-acquisition  radar  and 
target-tracking  radar  simulations  were  interfaced  with  the  SUT  at  RF.  The  projectile 
simulation  was  a  computer  simulation.  The  penetrating  aircraft  was  also  computer 
generated. 

3.3  Penetrating  Aircraft  (Closed  Loop)  (U).  The  AFEWES  is  capable  of  providing  RF 
simulations  of  as  many  as  four  penetrating  aircraft  for  most  threat  simulations.  Only  a 
single  aircraft  formation  was  used  during  this  test  program.  In  all  cases,  the  penetrator 
was  an  F-16C  aircraft  carrying  a  self-protection  jamming  pod.  The  radar  cross  section 
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(RCS)  simulated  for  the  penetrating  aircraft  used  modeled  F-16C  data.  The  RCS  was 
obtained  for  each  radar  from  data  matrices  based  on  the  individual  azimuth  and  elevation 
aspect  angles.  Generally,  a  special  tape  is  created  for  each  aircraft  flight  path  represented 
in  the  test.  AFEWES  elected  to  demonstrate  technology  that  would  be  used  in  the  later 
ADS-based  test  phases.  AFEWES  used  their  newly  developed  high  level  architecture 
(HLA)  interface  to  provide  the  AFEWES  facility  with  aircraft  position  and  attitude  based 
on  OAR  TSPI  data. 

3.4  JammEr  Technique  Simulator  (U).  The  JammEr  Technique  Simulator  (JETS)  is 
an  open-loop  simulation  which  can  be  used  as  a  surrogate  jammer,  a  cooperative  jammer, 
or  as  a  background  emitter  simulating  a  terminal  threat.  AFEWES  lacked  sufficient 
trained  operators  to  man  all  four  threats  at  the  same  time.  Therefore,  threats  were  manned 
in  pairs.  JADS  elected  to  use  the  JETS  to  simulate  unmanned  terminals  to  recreate  the 
OAR  RF  environment.  This  allowed  JADS  to  approximate  the  RF  load  on  the  jammer 
pod  processors  seen  in  the  OAR  to  better  recreate  threat  identification  and  response  times. 

3.5  AFEWES  Closed-Loop  Simulator  Instrumentation  (U).  Instrumentation  of  the 
AFEWES  closed-loop  simulations  was  provided  by  the  AFEWES  Test  Director  System 
(TDS).  The  TDS  consists  of  multiple  test  management  centers  (TMCs).  The  TMCs 
provide  instrumentation  and  control  of  the  individual  AFEWES  simulations  on  a  one-to- 
one  basis  by  the  following: 

•  optical  disks  for  mass  storage  of  raw  simulator  data 

•  digital  strip  chart  recorders 
digital  data  display 

an  online  printer  for  test  monitoring 

a  data  reduction  computer  for  processing  miss  distance  data 
video  tape  recordings  of  the  simulator  displays 

(U)  AFEWES  again  elected  to  augment  the  standard  data  collection  by  using  their  HLA 
interface  to  “publish”  the  data  elements  that  would  be  published  in  the  later  ADS-based 
phases.  This  allowed  JADS  to  avoid  the  cost  of  changing  the  data  analysis  tool  to  accept 
the  HITL  data. 

4.0  CORRELATION  METHODOLOGY  (U).  The  JADS  test  design  intended  to  use 
a  rigorous  statistical  comparison  of  results  from  different  test  environments  to  determine 
the  impact  of  using  ADS  in  testing  EW  systems.  The  statistical  comparison  chosen  was 
called  correlation.  As  JADS  test  design  progressed,  the  term  correlation  came  to  mean  a 
process  using  a  combination  of  analysis  and  statistics  to  compare  two  sets  of  data.  The 
basic  methodology  for  accomplishing  correlation  was  developed  from  a  CROSSBOW- 
funded  EW  test  results  correlation  study  conducted  by  Georgia  Technical  Research 
Institute.  JADS  first  articulated  the  methodology  in  their  May  1996  analysis  plan  for 
assessment.  The  method  centered  on  using  the  Smirnov  test  for  comparing  the 
distributions  of  the  data.  The  key  to  meaningful  correlation  is  not  the  test,  but  in  making 
sure  that  the  test  used  will  provide  meaningful  insight  into  the  data  for  decision  makers. 
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As  our  methodology  matured,  several  tests  were  considered  including  Kolmogorov- 
Smimov  (K-S),  Anderson-Darling,  Student  T-Test,  F-Test,  and  Chi-Squared  Test.  We 
selected  the  K-S  for  comparison  of  differences  in  distributions,  the  Student  T-Test  for 
comparison  of  differences  in  means,  and  the  F-Test  for  comparison  of  differences  in 
variances.  The  drawback  for  the  Student  T-Test  and  F-Test  is  the  assumptions  they  make 
about  the  underlying  population.  The  K-S  test  is  nonparametric  and  makes  no 
assumptions  about  the  underlying  population.  These  tests  gave  us  a  slightly  broader  view 
of  the  methodology  than  strictly  applying  the  K-S  test. 

(U)  The  process  used  was  straight  forward.  The  steps  are  listed  below. 

Step  1  -  Assemble  the  test  data  from  the  first  environment. 

Step  2  -  Assemble  the  test  data  from  the  second  environment. 

Step  3  -  Perform  qualitative  correlation  analysis. 

Graph  all  numbers  from  each  environment  as  a  function  of  run  number  to 
check  for  learning  or  other  future  “residuals.” 

Graph  the  raw  data  from  each  environment  as  a  relative  frequency 
histogram  and  experiment  with  different  interval  sizes  and  end  points  to  find 
the  most  meaningful  representation.  Make  an  initial  determination  about  the 
type  and  distribution  of  the  data. 

•  Qualitatively  compare  the  relative  frequency  histograms  between 
environments. 

■  Use  SUT  and  facility  knowledge  to  determine  if  additional  factors  are 
present  that  need  to  be  accounted  for  or  adjusted.  These  may  be  most 
evident  in  the  outliers.  Carefully  document  the  reasons  for  any  dropped 
data. 

•  Establish  the  new  baseline  data  from  each  environment  by  removing  or 
adjusting  all  data  affected  by  the  additional  factors. 

Step  4  -  Perform  quantitative  correlation  analysis  and  express  results  in  P-factors. 

Step  5  -  Interpret  results. 

Step  6  -  Document  results. 

(U)  JADS  intended  to  simply  publish  the  P-factors  and  allow  the  community  to  draw  its 
own  conclusions.  We  did  not  expect  perfect  correlation.  However,  examination  of  the 
data,  especially  in  light  of  the  concerted  effort  to  collect  repeatable  data,  caused  us  to 
question  the  viability  of  statistical  correlation  for  system  measures  of  performance  that  are 
highly  affected  by  operator  actions.  These  questions  apply  to  EW  systems  that  are 
required  to  work  against  human  operators.  Other  classes  of  systems  that  do  not  work 
against  human  operators  deserve  more  investigation  before  any  conclusions  can  be 
reached. 

5.0  RESULTS  (U).  More  than  267  runs  were  collected  in  two  weeks  of  testing  at 
AFEWES.  However,  all  objectives  were  not  met.  Although  internal  timing  measures 
were  collected,  we  discovered  post  test  that  all  of  the  instrumentation  was  not  time 
synchronized  and  these  measures  could  not  be  used  for  analysis.  An  additional  test  in  a 
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system  integration  laboratory  was  conducted  later  to  capture  these  timing  measures.  Data 
analysis  on  the  HITL  data  was  completed  and  several  conclusions  were  reached. 

•  The  data  were  sufficient  to  allow  JADS  to  create  one  ADS-based  test  environment 
and  execute  two  ADS-based  tests. 

The  data  collected  were  generally  sufficient  for  the  JADS  purpose  of  collecting  a 
significant  sample  of  data  for  characterizing  the  populations  of  the  ten  MOPs.  More 
data  would  have  increased  our  confidence  that  we  had  a  good  characterization; 
however,  more  data  would  have  made  the  statistical  tests  more  sensitive. 

•  The  data  collected  between  the  OAR  and  HITL,  while  generally  comparable, 
exhibited  very  weak  statistical  correlation.  While  several  factors,  including  threat 
simulator  variations,  contributed  to  this,  operator  actions  were  by  far  the  largest 
source  of  variation. 

The  data  collected  were  not  sufficient  for  verifying  and  validating  (V&V)  the  ADS- 
based  test  architecture.  The  OAR  and  HITL  data  were  useful  only  for  making 
general  comparisons.  Again,  statistical  correlation  between  the  ADS  and  baseline 
tests  was  very  weak  and  could  not  support  V&V.  As  expected,  the  operators  were  a 
significant  source  of  variation.  However,  we  found  evidence  to  suggest  that  the 
samples  may  not  fully  capture  the  real  variation  possible  within  the  population.  One 
of  the  threats  had  no  operator  variance  and  yet  displayed  high  variation  in  the  dry 
miss  distances  produced.  (Engineering  analysis  is  not  yet  complete;  however,  the 
preliminary  indication  is  that  slight  variations  in  the  flight  path  are  the  only 
identifiable  difference.)  JADS  had  to  rely  on  subject  matter  experts  to  V&V  the 
architecture. 

6.0  IMPLICATIONS  FOR  THE  EW  COMMUNITY  (U).  The  JADS  EW  Test  was 
focused  on  testing  a  RF  self-protection  jammer  against  threat  systems  employing  human 
operators.  Considering  this  narrow  problem,  the  following  are  our  observations  for  the 
EW  community. 

We  found  no  evidence  to  suggest  that  statistical  correlation  is  practical  for  measures 
of  performance  that  rely  on  human  operator  actions.  Certainly  there  may  be 
statistical  tests  that  could  be  performed  that  may  be  less  sensitive  to  high  variance 
and  outlying  data  points  that  human  operators  produced  in  our  test.  This  may  merit 
further  investigation.  However,  the  statistical  tests  we  performed  have  minimal 
utility  for  EW  decision  makers  given  the  current  infrastructure  and  affordable  sample 
sizes. 

•  For  similar  reasons,  we  found  no  evidence  to  suggest  that  statistical  correlation 
between  ADS  environments  and  OAR  test  data  for  V&V  of  the  environments  is 
practical.  Subject  matter  experts  are  a  better  source  for  V&V  of  ADS  environments. 

•  We  are  not  surprised  that  the  EW  community  has  difficulty  understanding  and 
articulating  military  utility  of  EW  systems  to  decision  makers.  Obtaining  repeatable 
results  in  a  simplistic  environment  proved  to  be  extremely  difficult  in  a  laboratory 
environment  such  as  AFEWES.  This  was  even  more  difficult  on  the  range.  We  do 
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not  believe  it  is  possible  to  understand  OAR  results  without  sufficient  laboratory 
testing  to  characterize  the  normal  operation  of  the  system  under  test. 
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